Several traffic anomaly detection methods have been proposed in the prior art:    Lakhina, M. Crovella, and C. Diot, “Mining anomalies using traffic feature distributions”, in Proceedings of SIGCOMM, August 2005;    A. Soule, K. Salamatian, and N. Taft, “Combining filtering and statistical methods for anomaly detection”, in Proceedings of IMC, October 2005;    X. Li, F. Bian, M. Crovella, C. Diot, R. Govindan, G. Iannaccone, and A. Lakhina, “Detection and identification of network anomalies using sketch subspaces”, in Proceedings of IMC, October 2006; and    F. Silveira, C. Diot, N. Taft, and R. Govindan, “Detecting correlated anomalous flows”, Thomson, Tech. Rep. CR-PRL-2009-02-0001, January 2009;and some techniques are now part of commercial products (Guavus NetReflex—http://www.quavus.com and Arbor Networks Peakflow—http://www.arbornetworks.com).
These methods have in common the ability to flag alarms for a variety of events that may be important to a network operations center (NOC) including attacks, flash crowds, link failures, and routing outages.
Once an alarm is raised, root cause analysis can be performed to know how to address the problem. Root cause analysis is usually left to network operators, who use their knowledge and intuition to analyze the traffic trace where the anomaly was flagged in search of events that can explain it. This manual process is both time-consuming and error prone. In a large ISP network with hundreds of links, the number of events that can trigger alarms may easily overload the NOC. Under such circumstances, the operator is likely to ignore alarms or never even deploy the detection system in the first place.
The root cause analysis problem involves two tasks: (1) identifying the traffic involved in the anomaly, and (2) classifying the anomaly according to the type of event that cause it (e.g., DoS attack, port scan, link failure).
Previous works have tried to address the root cause analysis problem by designing new detection methods with features that facilitate either identification (X. Li, F. Bian, M. Crovella, C. Diot, R. Govindan, G. Iannaccone, and A. Lakhina, “Detection and identification of network anomalies using sketch subspaces”, in Proceedings of IMC, October 2006 and F. Silveira, C. Diot, N. Taft, and R. Govindan, “Detecting correlated anomalous flows”, Thomson, Tech. Rep. CR-PRL-2009-02-0001, January 2009 or classification (A. Lakhina, M. Crovella, and C. Diot, “Mining anomalies using traffic feature distributions”, in Proceedings of SIGCOMM, August 2005).
Lakhina et al. (A. Lakhina, M. Crovella, and C. Diot, “Mining anomalies using traffic feature distributions”, in Proceedings of SIGCOMM, August 2005) proposed a method based on clustering of entropy residuals to classify the anomalies found by their PCA anomaly detector. Since these entropy residuals are an internal variable of the PCA detector, the main limitation of this approach is that it only classifies anomalies that are visible by PCA on entropy.
Li et al. (X. Li, F. Bian, M. Crovella, C. Diot, R. Govindan, G. Iannaccone, and A. Lakhina, “Detection and identification of network anomalies using sketch subspaces”, in Proceedings of IMC, October 2006) combined PCA with traffic sketches to develop Defeat, a detector that can also identify the traffic involved in the anomalies. Their solution is also restricted to the PCA detector, and it requires a modification to the original detector's algorithm, i.e., the aggregation of traffic into k-ary sketches.